Papers with reading comprehension models
Reasoning Over Paragraph Effects in Situations (D19-58)
Copied to clipboard
| Challenge: | a key component of reading a passage is the ability to apply knowledge gained from the passage to a new situation. |
| Approach: | They propose a benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. |
| Outcome: | The proposed model performs slightly better than randomly guessing an answer of the correct type, but is below the human performance of 89.0%. |
Machine Reading, Fast and Slow: When Do Models “Understand” Language? (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models of reading comprehension score highly on NLU benchmarks, but they are often 'read fast', i.e. rely on shallow patterns. |
| Approach: | They propose a definition for the reasoning steps expected from a system that would be 'reading slowly' they compare that behavior with five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. |
| Outcome: | The proposed model is compared with five models of the BERT family of various sizes, and compared using saliency scores and counterfactual explanations. |
Towards a more Robust Evaluation for Conversational Question Answering (2021.acl-short)
Copied to clipboard
| Challenge: | Conversational Question Answering (CQA) is a new form of NLP . it uses conversation history to extract the answer of the current question. |
| Approach: | They propose to use conversation history to evaluate models which can access the ground truth answers of previous turns at each turn of the conversation. |
| Outcome: | The proposed evaluation protocol severely limits the effectiveness of the proposed models in fully autonomous chatbots and leads to unsuspected biases in their behavior. |
Universal Adversarial Triggers for Attacking and Analyzing NLP (D19-1)
Copied to clipboard
| Challenge: | Using adversarial triggers, a model can produce a specific prediction . adversarial attacks are useful for evaluation and interpretation . |
| Approach: | They propose a gradient-guided search over tokens that finds short adversarial triggers that successfully trigger the target prediction. |
| Outcome: | The proposed algorithm finds short trigger sequences that successfully trigger the target prediction. |
Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Reading comprehension models often overfit to nuances of training datasets and fail at adversarial evaluation. |
| Approach: | They propose a method that introduces multiple points of confusion within the context and shows dependence on insertion location of the distractor. |
| Outcome: | The proposed methods improve robustness against adversarial evaluation but weak generalization to the source domain and new domains and languages. |
Zero-Shot Entity Linking by Reading Entity Descriptions (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to link entities to unseen entities require in-domain labeled data. |
| Approach: | They propose a zero-shot entity linking task where mentions must be linked to unseen entities without in-domain labeled data. |
| Outcome: | The proposed task can generalize to unseen entities without metadata or alias tables . the proposed system improves over baselines, including BERT, on a new dataset . |
Understanding Dataset Design Choices for Multi-hop Reasoning (N19-1)
Copied to clipboard
| Challenge: | Existing datasets that explicitly focus on multi-hop reasoning are lacking in learning multi-tasking. |
| Approach: | They propose to use sentence-factored models to solve multi-hop question answering tasks . they find spurious correlations in unmasked versions of WikiHop and HotpotQA . |
| Outcome: | The proposed datasets are used to test models on multi-hop question answering tasks. |
Single-dataset Experts for Multi-dataset Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has focused on training one network on multiple datasets to build a model that performs well on all of the training datasets and generalizes and transfers better to new datasets. |
| Approach: | They combine multiple reading comprehension datasets to build a multi-dataset question answering model with an ensemble of single-data set experts. |
| Outcome: | The proposed model outperforms baseline models in in-distribution accuracy and generalization and transfer performance. |
Learning with Instance Bundles for Reading Comprehension (2021.emnlp-main)
Copied to clipboard
| Challenge: | a study shows that training reading comprehension models assumes that the training instances are independent and identically distributed . however, this assumption can cause the learner to ignore distinguishing cues between related or minimally different questions . |
| Approach: | They propose to normalize question-answer scores across neighborhoods of closely contrasting questions and/or answers by adding a cross entropy loss term to the supervision signal. |
| Outcome: | The proposed methods show up to 9% absolute gains in accuracy on two datasets. |
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing reading comprehension benchmarks do not contain complex coreferential phenomena . obtaining questions focused on such phenomena is difficult because of lexical cues . |
| Approach: | They propose to use a crowdsourced dataset to examine the ability of models to resolve coreference among entities in Wikipedia paragraphs. |
| Outcome: | The proposed model performs significantly worse than humans on the reading comprehension benchmark . paragraphs and other longer texts typically make multiple references to the same entities . |